Manager repeatedly ignores developer's warning, resulting in $228k of incorrect transactions: 'The system did exactly what it was told to do'

Advertisement
  • A developer in a striped shirt sits at his desk, head in his hands, while a manager in a suit gestures demandingly at a computer monitor displaying analytics.
  • One time my manager was on my case about a client integration delay. The issue wasn't on our side, the partner we were integrating with hadn't properly completed their error handling.
  • I told him we needed to wait until they finalized it so we could correctly map response codes and avoid processing issues. He got impatient and escalated. He called the partner's manager, who then looped in their tech manager. The three of them agreed that anything not explicitly marked as "success" should be treated as failed and retried up to three times if failure continues.
  • He came back and relayed the decision to me. I raised concerns again. Since their system was still in active development, I warned that if they returned unexpected statuses, we could run into serious problems.
  • The response was the classic: I've spoken to their manager and tech lead. Implement it as discussed. He even looped in CTO and CEO in the email to shift delay blame.
  • So... Not Successful → wait 10 seconds → retry Repeat if not successful → stop after attempt 3 At that point I was exhausted. I implemented it exactly as instructed because clearly my concerns weren't being taken seriously.
  • Everything ran smoothly for 6 days. Then the provider started returning HTTP 403 responses with HTML bodies instead of JSON. The system did exactly what it was told to do.
  • Long story short, 228K was incorrectly consumed across about 90 customers. The provider was actually processing the transactions successfully, but they had pushed a broken update on their side. Our system interpreted the unexpected 403 responses as failures and retried, causing duplicate processing.
  • Saturday afternoon, I'm at home having drinks when my phone starts blowing up. Manager in full panic mode. I log in from my home PC, pull the logs, and send him a detailed breakdown of the issue and the incorrect dispatches.
  • Then I forward the email thread where I had clearly outlined this exact risk including his reply telling me to execute and stop delaying and I go back to drinks... CTO: Royally p ed. CEO: *To manager* "Your staff seem to know something you don't"..
  • Manager: *crickets* Issue was eventually fixed and provider took blame partially and losses were split.. What have you been pushed to do knowing it will backfire in case something goes wrong and it does spectacularly and you just have to sit and watch the fireworks?
  • breezy_owl_1989 Sounds like you had a great day today!
  • nakedmogash I do this every day. Thankfully I'm a designer so the only results are bad posters and videos
  • hymie0 We decided to get into the mass email campaign business. When I asked if we were becoming spammers, I was told in so many words that business decisions are not made based on my personal politics.
  • Three days later, I'm cleaning thousands of email bounces off our server. Two days after that, I'm building a new email server with a new IP address because every spam list had our ip listed. At least my boss had the decency to be contrite about it.
  • ThriveBeyond-C... Haha you narrated it in the best way possible

Tags

Scroll Down For The Next Article